feat: store event bloom filter in DB #473
Open
+1,526
−51
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jbcaron
added
feature
jbcaron
force-pushed
the
feat/bloom-filter
branch
from
ca2b119 to
42e4668
Compare
jbcaron
force-pushed
the
feat/bloom-filter
branch
from
42e4668 to
f5ef319
Compare
jbcaron
force-pushed
the
feat/bloom-filter
branch
from
de25cf7 to
cff833c
Compare
Trantorian1
approved these changes
Feb 3, 2025
| #[cfg(test)] | ||
| #[test] | ||
| fn test_column_all() { | ||
| assert_eq!(Column::ALL.len(), Column::NUM_COLUMNS); |
There was a problem hiding this comment.
Is this really useful since NUM_COLUMNS is already defined as Self::ALL.len();?
crates/madara/client/rpc/src/versions/user/v0_7_1/methods/read/get_events.rs
Outdated
Show resolved
Hide resolved
crates/madara/primitives/bloom_filter/benches/bloom_benchmark.rs
Outdated
Show resolved
Hide resolved
Comment on lines
+166
to
+182
| fn bench_bloom_filters(c: &mut Criterion) { | ||
| let test_data = generate_test_data(TEST_DATA_SIZE); | ||
|
|
||
| // Test different hash functions | ||
| let hashers = [ | ||
| get_hasher_benchmarks::<DefaultHasher>("DefaultHasher"), | ||
| get_hasher_benchmarks::<AHasher>("AHasher"), | ||
| get_hasher_benchmarks::<XxHash64>("XxHash64"), | ||
| ]; | ||
|
|
||
| for hasher in &hashers { | ||
| (hasher.sequential_insertion)(c, &test_data, hasher.name); | ||
| (hasher.parallel_insertion)(c, &test_data, hasher.name); | ||
| (hasher.lookup)(c, &test_data, hasher.name); | ||
| (hasher.parallel_lookup)(c, &test_data, hasher.name); | ||
| } | ||
| } |
There was a problem hiding this comment.
Really cool testing structure!
| /// where: | ||
| /// - k is the number of hash functions | ||
| /// - p is the desired false positive rate (0.01) | ||
| /// Reference: [Bloom, B. H. (1970). "Space/Time Trade-offs in Hash Coding with Allowable Errors"](https://dl.acm.org/doi/10.1145/362686.362692) |
Comment on lines
+8
to
+14
| pub const HASH_COUNT: u8 = 7; | ||
|
|
||
| // Number of bits per element in the Bloom filter. | ||
| // The value 9.6 is optimal for a false positive rate of 1% (0.01). | ||
| const BITS_PER_ELEMENT: f64 = 9.6; | ||
|
|
||
| pub const FALSE_POSITIF_RATE: f64 = 0.01; |
There was a problem hiding this comment.
These constants appear several times in the codebase, maybe we want to define them only once and re-use them?
jbcaron
force-pushed
the
feat/bloom-filter
branch
from
7b985e6 to
56a51cc
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request type
Please add the labels corresponding to the type of changes your PR introduces:
What is the current behavior?
Currently, the event pagination mechanism does not correctly handle the continuation token
Additionally, event filtering does not leverage a bloom filter, which could improve performance by reducing unnecessary block lookups.
Resolves: #NA
What is the new behavior?
Integration of
mp-bloom-filterfor event filtering:mp-bloom-filtercrate to improve filtering efficiency.Optimized event retrieval:
take(chunk_size + 1)in the filtering process to determine whether the current block has more events.Does this introduce a breaking change?
Yes
Other information